Papers with Long short-term memory
How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? (2020.acl-srw)
Copied to clipboard
| Challenge: | Long-term memory (LSTM) networks are capable of encapsulating long-range dependencies . but simple recurrent networks (SRNs) have been less successful at capturing long-term dependencies and loci of grammatical errors in an unsupervised setting. |
| Approach: | They propose a new architecture that incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons. |
| Outcome: | The proposed architecture shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks. |
Binarized LSTM Language Model (N18-1)
Copied to clipboard
| Challenge: | Long short-term memory (LSTM) language models are widely used for automatic speech recognition and natural language processing (NLP) however, they are limited by the word embedding layer. |
| Approach: | They propose to encode words into binary vectors and use binarized LSTM parameters to achieve high memory compression. |
| Outcome: | The proposed model achieves 11.3 compression ratio without loss of performance and 31.6 compression ratio with acceptable performance degradation. |